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ABSTRACT 



The assessment and diagnostic procedures of a 
comprehensive accountability plan were applied to several elementary 
schools from a large midwestern state. Pretest and posttest Word 
Knowledge and Reading scores from the Primary II Metropolitan 
Achievement Test administered in 1970-71 to third-graders were used. 
These data were used to compute student Development Indices (SDIs). 
The SDIs were related to school process variables. Application of the 
diagnostic procedures resulted in the identification of the total and 
unique contributions of the process variables, individually and in 
combination. The implications of the assessment and diagnosis for 
corrective action are discussed, (Author) 
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AN EXAMPLE OF THE APPLICATION OF THE ASSESSMENT AND 
DIAGNOSTIC PROCEDURES OF A COMPREHENSIVE ACCOUNTABILITY PLAN**" 

Gary L. Marco 
Educational Testing Service 

A few days ago I took a boat ride Into bayou country and learned the 
jiifference between a marsh and a swamp. ^fF~fli»i»fflff" ^-1i .i t n mm hih 1s dl ^^^'^p nt' 
from a swamp In the same way that a prairie Is dlffer^nit from a forest. If 
I asked a native what the difference between a marsh and a swamp was, I'm 
sure he wouldn't think of this .analogy. He*d probably say something like, 
"Well, son. If you go out wandering and get lost, you'll know you're In a 
swamp • " 

Today I shall take you through the marshland of an accountability 
plan. We'll look at a tew of the prominent features having to do with 
assessment and diagnosis. I hope that the marshland doesn't turn Into a 
swamp before we finish. To point you on your way I want first to give you 
a brief Introduction to the accountability plan I have In mind. After that 
we'll look at an example that shows how the assessment and diagnostic 
procedures of the plan can be applied. 



Presented at the meeting of the American Educational Research 
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The Accountability Plan 

The accountability plan was designed for elementary and secondary 
schools, and focuses on the school as the basic unit of accountability. 
It Includes many features of Dyer's student- change model, which was 
described In the Phi Delta Kappan (Dyer 19; 

The plan consists of three phases: assessment, diagnosis, and 
corrective action «L__In_the_jissess^ phase the achievement of students 
Is measured at two or more points In time. Data based on students who 
have stayed In the same schools over the last two data collection points 
are used to derive measures of school effectiveness. I should point out 
that such longitudinal samples may be small In schools having a highly 
transient student population. (If' Is Interesting that schools often do 
not have an accurate Indices of student stability, although they do have 
Indices of student mobility. Since thd same students who transfer in 
often transfer out, a student mobility Index can give a very misleading 
estimate of the number of students who have remained in the school over 
a given period of time.) The accountability plan assumes that assessment 
data are available over a two-year period. However, the plan can be 
adapted to longer or shorter periods of time. And, in fact, the 
Illustrative data that I will use covers only a one- year period. 

Key to assessment is the computation of one or more indices of student 
development for each school. The Index is a number that expresses the 
extent to which students at a given grade level in the school have 
progressed in a particular area of achievement over a specified length of 
time. The index takes into consideration where a student was and where 
student stands now, but as you will see it Is not the usual gain measure. 
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The student development indices serve to identify schools in which 
students are performing relatively poorly. It is these schools that need 
to take corrective action. However, before corrective action is taken, 
the school staffs need some guidelines to go on. It is the purpose of the 
diagnostic phase of the accountability plan to provide these guidelines by 
identifying possible causal variables that might be working at the school 
level. 

In the diagnostic phase tHe'various variables that might be related 
to student development must be sorted into those that corrective-action 
plans might focus on and those that the plans can ignore. These variables 
are called process variables . They refer to any characteristic of the 
school program, facilities, students, staff, or community which might 
reasonably be expected to relate to student development and over which the 
school can exercise some, degree of control. Diagnosis proceeds in two 
directions: (1) the statistical analysis 6f the available data to show 
the relative contributions of the various process variables to student 
performance; (2) case studies of schools having high and low indices of 
student ^development. The case studies are particularly useful for 
uncovering important process variables that were unmeasured to begin with. 

In the last phase of accountability, corrective action , the schools 
having low indices of student development are required to produce 
correction-action plans. These plans would ordinarily focus on variables 
that were identified in the diagnostic phase as important. It is the 
diagnostic and corrective-action phases that make the accountability plan 
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different from the usual school assessment plan, which stops with an 
evaluation of student performance. Accountability Is thus defined In 
terms of the responsibility of school personnel In designing, 
Implementing, and evaluating corrective action* 

So far, I have given you a brief but, I hope, adequate description 
of the essential components of a comprehensive accountability plan* In 
the time remaining I shall concentrate on pr^^^'4Tii^«»^-TTftM--4=fK^,ffTnpMtp fhe 
Indices of student development and procedures used to assess the Importance 
of process variables* 

Sample 

The procedures will be applied to Metxopolltan Primary II Reading 
Achievement Test data from six elementary schools* The test was 
administered to third-graders In the fall and spring of the 1970-71 
academic year* For demonstration purposes the Word Knowledge and Reading 
scores will be used* Pretest and post test means for those students who 
took tests In both the fall and the spring are shown In Table 1* Note 
that even though Schools D and E had similar pretest means. School E had 

Insert Table 1 about here 

much higher posttest means* Moreover, School F had lower pretest means 
than School C but a higher Reading posttest mean* We would expect the 
Indices of student development to reflect the superior performances of 
Schools E and F* 
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Assessment 

The stiident development indices for the schools are really mean 
individual residual scores and are not particularly difficult to compute. 
First the regression coefficients were estimated for predicting the 
Reading posttest and Word Knowledge posttest scores, respectively, from 
both Reading and Word Knowledge pretest scores* The scores used were 



those of all 238 third-graders having pretest ana postre8r**^ofes^ 

The models used here did not employ higher-order terms such as the 
squares of the pretest scores, but there is no reason why such terms 
could not be added if the relationships were not assumed to be linear. 
In actual practice, the coefficients for a complete second-order model, 
employing first- and second-degree terms, would probably be estimated 
and tested for statistical significance. Cross-validation would also 
be used to ensure that the coefficients would work on new samples. 
The coefficients for the two regressions were: 

A 

R ^ = 13.38927 + 0.46532 WK + 0.39521 R , and 
post pre pre 

WK ^ - 13.74727 + 0.52659 WK_ + 0.35294 R . 
post pre pre 

The mean individual residual is simple to calculate once the 

regression coefficients have been estimated* The expected Reading score 

A 

(R ) for a school is computed by substituting the Word Knowledge and 

POO k 

Reading pretest means for the school into the first regression equation. 
Thus, the expected Reading score for School A is 13.38927 + 0.46532 
(46.2308) + .39521 (41.0256), which equals 51.1151. This value is the 
expected Reading posttest performance based on the average performance 
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of students In the six schools. The mean Individual residual is equal 
to the post test mean for the school minus the expected mean. In the 
case of School A, this Is 46.8718 - 51.1151 = -4.2433. The number Is 
negative^ which Indicates that the performance of the average student 
In this school In reading was below the average for all students. 

The 12 mean Individual residuals that were computed along with the 
pOBt tcot moanc a n d expq o tc i perf or m ances are shown In Table 2 for both 

Insert Table 2 about here 

Reading and Word Knowledge. These mean Individual residuals are displayed 
graphically In Figures 1 and 2, which are In your handout. You may note 
that the superiority of Schools E and P Is quite apparent. On the other 



Insert Figures 1 and 2 about here 

handy Schools A and D are below the line of prediction and would be 
singled out for special attention. (Of course^ It would be useful to 
establish confidence limits for the predicted values to ensure that the 
deviation from prediction did not arise from sampling fluctuation.) 

Before turning to procedures used for diagnosis, I should point out 
that my discussion of the assessment phase Is incomplete. Quirk (1973) 
presents a more complete discussion. 
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Diagnosis 

It Is not enough to know that the students In some schools are doing 
well and others not so well In Reading and Word Knowledge. Something must 
be done to Improve performance* The diagnostic component of the account- 
ability plan Is designed to help staff know what has worked within the 
system. Process Information from the various schools Is assembled , reduced , 

and r elated to student performance* At this stage the relationships are 

only correlational 9 not causal* Later, when corrective-action plans are 
formulated y the assumption of causation must be made* Identifying Important 
process variables Is much like trying to find out the cause of your stomach- 
ache after having breakfast at Brennan's, cocktails at the Top of the Mart, 
and dinner at Antolne*s* 

Initial Data Analysis 

In this small-scale tryout of the accountability procedures only five 
process variables were used: pupil/teacher ratio (P/T), percent of 
teachers with five or more years experience (%EXP) » the percentage of 
non-white students (%N-W)9 the K-12 current operating expense per pupil 
($/PUPIL), and the number of third-graders (N)* You may argue that some 
of these are not under the control of the school and thus are not good 
choices for process variables* However » of these variables > most can be 
changed at the school district level If not the school level* If a 
large number of variables had been available > factor-analytic methods could 
have been used to reduce the data to the desired number of variables* 



There are two common kinds of analyses that are less than informative 
than the one I propose. The first is a multiple regression analysis of 
individual pupil scores with the pretest scores and the process variables 
used as independent variables • This analysis ignores among-school 
variation. 

The second is a regression analysis using (a) the school Mean 
Residuals obtained by partialing out the student pretest scores from 
the student post test scores and (b) the process variables. This analysis 
utilizes only among-school variance and ignores within-school variance. 
The analysis I shall illustrate makes use of both among-school and 
within-school variation. 

There is a third analysis that some might use. It is an analysis 
of the variance associated with the process variables and the among-school 
variance. This kind of analysis is wrong if these variables are treated 
simultaneously in a regression analysis. The approach I suggest looks at 
among-school variance and the variance associated with process variables 
sequentially . The key to this analysis is obtaining the among-school and 
wlthln-school sums of squares before looking at the process variables. 

The first step in the analysis is to eliminate the influence of the 
Reading and Word Knowledge pretests. This is done by treating the pretests 
as concomitant variables in a multiple regression analysis. If you will 
refer to Table 3 you can see that in the illustrative data the pretests 
accounted for more than half of the total sum of squares for Word Knowledge 



Insert Table 3 about here 
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and Reading* The next step Is to compute the sum of squares associated 
with differences among the school Intercepts » assuming common wlthln- 
group regression coefficients. Finally , the sum of squares accounted for 
by differences In the regression coefficients for the pretests Is computed* 
The results of these analyses are also shown In Table 3. Note that the 
hypothesis of common regression coefficients Is rejected at <>^«05 for 



If there were no differences among the regression coefficients or 
Intercepts, It would not be worthwhile to search for school process 
variables that account for among-school variance. Rather one would have 
to look within the schools to Identify important process variables 
associated with either classes or Individual students. 

Here the statistical tests permit us to continue. We proceed to 
look at the contributions of the five process variables. Table 4 shows 
how the schools stood on these variables. Computationally, the five 



process variables were simply substituted into the regression program in 
place of the school dummy variables. A forward-selection stepwise 
regression procedure was then used to order the process variables in terms 
of their contributions to the school int^jrcept sum of squares. The results 
are given in Table 5. The error terms for the F-tests were the residual 
mean squares from Table 3. 




Insert Table 4 about here 



Insert Table 5 about here 



ERIC 



-lo- 



in the case of both Word Knowledge and Reading , the same three process 
variables accounted for practically all of the variance associated with 
differences In school Intercepts. These were the percentage of non-white 
students 9 the ntimber of third graders , and the percentage of teachers with 
five or more years of teaching experience. 

We note In passing that If the degrees of freedom^Qr_schools had 



exceeded the number of process variables » It would have been possible to 
test the residual school Intercept variance to see If adding additional 
process variables would have been very profitable. You may note that 
the variance associated with the school regression coefficients was not 
analyzed. In a full-scale analysis^ the contribution of the process 
variables to this variance would also be Investigated. 

Newton-Spurrell Procedure 

One final step remains to complete the analysis. We need a measure 
of the Importance for each of the three process variables selected by 
stepwise regression. The proportion of sum of squares accounted for as 
each variable Is added In the stepwise solution Is not a good measure. 
Since the variables are correlated with each other ^ changing the order 
In which the variables are added changes the sum of squares accounted for. 
The procedure suggested for use In the accountability plan Is the one 
described by Newton and Spurrell (1967) and used by Hayeske In reanalyzing 
tbe Coleman data (Mayeske et al. $ undated) . This procedure partitions 
the variance associated with a given variable Into the portion accounted 
for by the variable uniquely and the portions accounted for by the variable 
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In combination with other variables. To do this partitioning, we must 
obtain the proportion of variance accounted for in the. Reading and Word 
Knowledge posttest scores by each of the six possible combinations of 
the three process variables. These proportions are listed in Table 6 
in the column labelled "Total Contribution." >t contribution 

Insert Table 6 about here 

is simply the squared multiple correlation. The unique contributions 
were computed from the total contributions. If you will look at the 
Venn diagrams in Figure 3, you will note that the unique contribution 

Insert Figure 3 about here 

of a given variable is simply the total contribution of all three 
variables minus the total contribution of the other two variables. 
Thus, the unique contribution of % Non-White in Figure 3a is .0291. 
Once the unique contributions of the individual variables are known, it 
is relatively easy to find the other unique variances. For example, 
unique contributions of % Non-White and N combined is what is left of 
the total contribution after the total contribution of the percentage 
of teachers with five or more years of teaching experience and the 
unique contributions Of the other two variables are taken out. 
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So much for the computations. If the marshland has turned Into a 
?^Tamp fcr some of you, maybe we can start finding our way out now. We 
jilL ' to select the variables we might like to focus on in a corrective- 
action plan* We should probably concentrate on those combinations that 
have (a) high unique contributions relative to their total contributions 
and (b) high total contributions* Where do these rules lead us in the 
present case? We note in Table 6 that for Word Knowledge % Non-White 
had a reasonably high total contribution and that its unique contribution 
was nearly half of its total contribution* So % Non-White seems to be 
an important variable in the case of Word Knowledge* For Reading the 
unique contribution of % Non-White is nearly equal to its total 
contribution, but its total contribution made up only about half of the 
total contribution of all three variables* However, the unique 
contributions of the combinations that have larger total contributions 
than % Non-White are so low that % Non-White seems to be the choice here 
also* 

What does it mean for a corrective-action plan to focus on % Non- 
White. We must first note whether the regression weight for % Non-White 
Is positive or negative* Here % Non-White had negative weights for 
estimating both Word Knowledge and Reading posttest scores. However, it 
would be misleading to treat % Non-White as a causal variable, even if 
the school district could decrease the percentage of non-white students. 
It is the job of technical staff to decide why % Non-White was related 
to student development in Word Knowledge and Reading. At this point 
the diagnostic procedure becomes more of an art than a science* On the 
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basls of what Is known about the schools. It may be decided that the 
schools having high proportions of non-white students have poor physical 
facilities or that they have few Innovative reading programs. Somehow 
real underlying causes must be Identified* The corrective-action plan 
would then focus upon the hypothesized causal variables* The Important 
point here Is that the diagnostic analysis suggested the direction In 
which to look for a causal Influence but did not Indicate necessarily 
what the causal Influence was* 

Once possible underlying causal variables are Identified, the 
diagnostic phase Is finished* Of course, case studies would be necessary 
If the process variables accounted for only a small portion of the among- 
school variance* In the accountability plan the results of the diagnosis 
would be written up In the form of guidelines* And, on the basis of the 
guidelines, corrective-action plans would be formulated* A school's 
corrective-action plan would focus on the causal variables thought to 
be underlying the important process variables* It should be clear from 
this example that analysis carries one only so far in diagnosis* 
Judgment must take over when causation is inferred and the corrective- 
action plan is formulated* 

We started out today walking through a marshland* I hope the 
features I have Illustrated will help us make our way to dry ground* 
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Table 1 

Pretest and Posttest School Means 
School A School B School C School D School E School F 





(N-39) 


(N-56) 


(N-29) 


(N-52) 


(N-28) 


(N-34) 


Wd. I'sian. Pre 


46.2308 


50.3214 


62.7241 


50.4615 


50.2500 


57.1765 


Reading Pre 


41.0256 


47.5714 


60.6552 


45.3462 


45.7857 


57.2647 


Wd.. Know. Post 


48.3590 


58.9643 


68.2759 


53.0769 


59.6071 


67.9412 


Reading Post 


46.8718 


57.2679 


64.3103 


51.3077 


58.7143 


68.8235 



Table 2 

Expected Performances and Residuals for Each School 



School A School B School C School D School E School F 





(N-39) 


(N-56) 


(N-29) 




(N-28) 


(N-34) 


Wd. Know. Post 


48.3590 


58.9643 


68.2759 


53.0769 


59.6071 


67.9412 


Wd. Know. Expected 


52.5715 


57.0359 


68.1848 


56.3243 


56.3680 


64.0668 


Wd. Know. Residual 


-4.2125 


1.9284 


0.0911 


-3.2474 


3.2391 


3.8744 


Reading Post 


46.8718 


57.2679 


64.3103 


51.3077 


58.7143 


68.8235 


Reading Expected 


51.1151 


55.6055 


66.5476 


54.7913 


54.8666 


62.6262 


Reading Residual 


-4.2433 


1.6624 


-2.2373 


-3.4836 


3.8477 


6.1973 



•The expected perfomance waa computed by substituting a school's 
pretest scores into the regression equation for all sdiools. The 
residual Is the difference betveen the posttest mean and the 
expected performance. 
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Table j 

Analysis of Variance of Posttest Scores 



Source of Variation 


S.S. 


df 


M.S. 


F 


Pr 






WORD KNOWLEDGE 








Total (Centered) 




32158.4244 


237 








Pretests 




19554.6634 


2 


9777.3317 


231.712 


<.01 


School Intercepts 




2500.3699 


5 


500.0740 


11.851 


<.01 


School Regression 


Coefficients 


820.2808 


10 


82.0281 


1.94h 


<.05 


Residual 




9283.1103 


220 












READING 










TO tai v^en t er ea ) 




34547.1765 










Pretests 




19228.4306 


2 


9614.2153 


196.603 


<.01 


School Intercepts 




3577.9198 


5 


715.5840 


14.633 


<.01 


School Regression 


Coefficients 


982.4469 


10 


98.2447 


2.009 


<.05 


Residual 




10758.3792 


220 


48.9017 










Table 4 












Standings of Schools on 


Process 


Variables 






School 


$ /PUPIL 


X N-W 


ill 


% 5+ EXP. 


N 




A 


857 


74 


29.0 


52 


93 




B 


676 


1 


30.6 


67 


66 




C 


64A 


2 


22.8 


80 


32 




D 


714 


99 


21.7 


52 


66 




E 


878 


26 


20.9 


60 


36 




F 


893 


1 


24.7 


57 


38 





Table 5 

Selection of Process Variables by Step-Wise Regression (Forward Method) 



doui^cb vax^ iun 


s.s. 


df 


M.S. 


I 


Pr 




WORD KNOWLEDGE 










School Intercepts 


2500.3699 


5 








X N-W 


1901.4511 


1 


1901.4511 


45.062 


<.01 


N 1 N-W 


336.4170 


1 


336.4170 


7.973 


<.01 


% 5+ EXP. 1 N-W, N 


200.9094 


1 


200.9094 


4.761 


<.05 


9 / rUrlL | n-W, N, lUUr. 


44 73Q3 


1 


44.7393 


1.060 


>.10 


P/T 1 N-W, N, E3tP., $ 


16.8531 


1 


16.8531 


0.399 


>.10 




READING 










School Intercepts 


3577.9198 


.5 








X N-W 


2039.5025 


1 


2039.5025 


41.706 


<.01 


X 5+ EXP. 1 N-W 


919.2727 


1 


919.2727 


18.798 


<.01 


N 1 N-W, EXP. 


549.9113 


1 


549.9113 


11.245 


<.01 


$ / PUPIL 1 N-W, N, EXP. 


58.4553 


1 


58.4553 


1.195 


>.10 


P/T 1 N-W, N, EXP. ,. $ 


10.7780 


1 


10.7780 


0.220 


>.10 




Table 6 








1 



Total and Unique Contributions of Various Combinations 
of Three Process Variables 



Word Knowledge Reading 



Variable 


Total 
Contribution 


Unique 
Contribution 


Total 
Contribution 


Unique 
Contribution 


X N-W 


.0591 


.0291 


.0590 


.0542 


N 


.0454 


.0125 


.0458 


.0159 


X 5+ EXP. 


.0166 


.0062 


.0036 


.0318 


X N-W, N 


.0696 


.0176 


.0697 


.0278 


X N-W, X 5+ EXP. 


.0633 


-.0049 


.0856 


-.0052 


N, X 5+ EXP. 


.0467 


-.0020 


.0473 


-.0303 


X M-W, H, % 5+ EXP. 


.0758 


.0173 


.1015 


.0073 
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Expected Posttest -'terd Knov/ledge Mean 



Fig* 1* Posttest Word Knowledge means plotted 
against expected performance* 
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(a) Word Knowledge (b) Reading 

Fig. 3* Unique contributions of combinations of three variables 
to the variances of Word Knowledge (a) and Reading (b) 
posttest scores* 



